Concerning the security problem caused by the singleness of rotation factor and grouping in FRactional Fourier Transform (FRFT) image encryption algorithm, a novel encryption method, namely chaotic dynamic random grouping and random modulating FRFT rotation factor, was proposed. Three chaotic systems were adopted in this process. Firstly, the keys of subsystem 1 were utilized to realize the nondestructive encryption on plaintext and obtain the first cipher text. Then subsystem 2 was utilized to realize the dynamic grouping of FRFT, and the rotation factor of FRFT was randomly modulated by using subsystem 3. After that, the FRFT was utilized to realize the second encryption on first cipher text. The test results on adjacent pixels correlation and pixels change rate show that the method can effectively resist statistical attack and differential attack. Compared with destructive method, in the first encryption, the real-time and reduction test show that the decrypted time of nondestructive method decreases by 52.5% and the similarity of decrypted image improves by 4.5%; thus, the denoising process is avoided and the system overhead is reduced. Security test shows that the information entropy increases by 1.7% and the ability to resist exhaustive attack increases by 10 ³⁶³⁵ times, compared with simple modulate FRFT rotation factor algorithm. The results show that the proposed method is better than the chaotic key simple modulating FRFT rotation factor algorithm in real-time, reduction, security and so on.